chdkfandomcom-20200222-history
CHDK/Camera RAM memory usage
= Camera RAM memory usage = Disclaimer: This was originally written by me (fudgey) on January 2009, and it may very well contain errors. I'm not an expert in these things, I'm just documenting what I'm finding out. Please fix or report any errors you may find. Some things will end up being obsolete sooner or later unless fixed. Most tests were done using a slightly modified trunk 665 (CHDK 0.9.0) for a570is 1.00e, compiled on gcc 3.4.6. Both compiler version and camera platform affect the RAM requirements significantly. Even though I have quoted fairly accurate numbers, they should be taken as approximate guidance only! For example, autobuilds use gcc 4.x.x, which produces slightly smaller code. Introduction CHDK supports dozens of different Canon cameras, not all of which were created equal. One way they differ from each other is the amount of RAM memory available to CHDK. The camera is much like any other computer and CHDK uses the same RAM used by much of the camera's original functionality. The lack of RAM is a significant problem in some cameras, causing seemingly random crashes if too many features are enabled. Fortunately many things can be done to conserve RAM, read on. Things in RAM could be divided to 5 distinct groups: # Static Canon variables: Always occupying a fixed amount of RAM. Nothing we can do about this part. # Dynamic Canon variables: Allocated on-demand whenever Canon firmware needs RAM for something. If CHDK allocates too much RAM, it's possible that Canon's code will not get the RAM it requires and the camera may crash (typically it freezes and shuts down after a while with lens out). TODO: how much continous free RAM do our cameras require to be stable in everyday shooting? # CHDK program code: Always there if CHDK is running. This part can be made smaller by disabling features in CHDK source code and then building and installing a customized version of CHDK. Note that Canon program code is not in RAM, it's in ROM. CHDK can't be put into ROM. # CHDK static RAM requirements: Always there if CHDK is running. Becomes smaller by disabling some CHDK features. The developers have lately been optimizing the most memory hogging features by making them allocate RAM dynamically. # CHDK dynamic RAM requirements: Required by CHDK while a certain feature is in use. This part can be smaller by not using some CHDK features that require RAM. If CHDK fails to allocate RAM, the feature probably fails to work. Unfortunately there is no low RAM warning or memory allocation failure error reporting to the user in CHDK currently. Note that if you just have enough memory to start a feature, it's possible that the camera crashes soon afterwards if Canon code is then unable to allocate memory. Developers, see CHDK Coding Guidelines for related information. Note there is experimental support for an alternate memory allocation system, which can increase the amount of RAM available to CHDK and reduce the amount of the original Canon heap used. See ExMem - enabling and loading CHDK into extended memory. How to make CHDK use less RAM Users have varying needs. This is why we don't want CHDK autobuild releases to have a limited set of features for any given camera, not even for ones lower on RAM than others. There are two ways to limit RAM usage: configure and use CHDK with RAM savings in mind (easy as eating pie, any user should be able to do this) and/or build your own CHDK (with current tools this is easier than you may fear, see but if CHDK-Shell scares you, feel free to request someone to build you a customized version on the discussion forum http://chdk.setepontos.com/index.php). Save RAM by configuring CHDK Features that can be disabled by configuring CHDK (some of these RAM requirements may vary among cameras especially if LCD display resolution differs from a570is, but these sizes should give you an idea of the order of significance): * Edge overlay (260 kB) :Note: Edge overlay reserves RAM when you first use it (i.e. enable it in CHDK menu Edge overlay page and then shoot a photo). There is an option for freeing edge overlay RAM. Cameras with very low memory may be completely unable to start edge overlay unless using a stripped down custom build and a low feature set configuration. * Zebra (110 kB) :Note: If you have a camera with enough RAM, you can make Zebra behave nicer by building your own CHDK with '#define ZEBRA_CANONOSD_BORDER_RESTORE 0' (instead of 1) in core/gui_osd.c. This may one day become a menu option. This increases Zebra RAM usage to ca. 170 kB but it will then restore the entire Canon OSD, including Canon grid and focus square instead of just the top/bottom borders that hold exposure information. :Note: Zebra RAM is reserved when zebra is drawn for the first time after power-on. To free zebra RAM, you must disable it in CHDK menu Zebra settings page and then power off the camera. As of build #683, zebra memory is allocated when the zebra needs to be displayed, and freed as soon as it is not needed. * DNG output (requirement depends on the size of your badpixel.bin, ca. 26 kB with about 9000 bad pixels). This is automatically loaded when you enabled DNG, and unloaded when DNG is disabled. * Language file other than default built-in English (12 kB). Never select another language (remove CHDK/CCHDK.CFG to be sure you haven't). * Menu font other than default built-in small system font (e.g. 12 kB). Never change the menu font from default (remove CHDK/CCHDK.CFG to be sure) * CHDK menu symbols icons (e.g. 7 kB). Disable symbols in CHDK menu OSD page (TODO: is that where it's disabled and does it help? Is reboot required to free RAM? I didn't check). * Scripts: To conserve RAM, do not have a script loaded when you don't need a script. To unload the script (and revert back to built-in default script), load an empty script file and power off the camera to free RAM. There is an empty script in the full CHDK distribution, in CHDK/SCRIPTS/examples/default.bas. When not running, a script requires approximately as much memory as it's size on the memory card (for example about 4 kB for MDFB-080914.bas motion detector script). Scripts may require more RAM when they are running. For Lua only sky and camera resources are the limit for RAM usage! There are already Lua scripts which cannot be used on all cameras or configurations because of their RAM requirements. FBonomi's sunset timelapse script is one of these and fudgey's enhanced version of it even more so (I think it requires more than 150 kB free RAM to work). Lua scripts can monitor and manage their memory use with the collectgarbage function. * Custom curves: In build 690 and later enabling curves uses 8kB (for custom) or 4kB (for +1,+2 and auto). Save RAM by making a customized build Features easily disabled by modifying buildconf.inc (e.g. change PT_GAME_REVERSI=1 to #PT_GAME_REVERSI=1) and recompiling: * Custom curves (19.9 kB). :Note: As of changeset 690, curves allocate memory only when the option is used. Disabling them at compile time only saves a further 3kB. * Games (14.1 kB total): ** Reversi (3 kB) ** Sokoban (4.1 kB) ** Connect4 (4.5 kB) ** Mastermind (2.5 kB) * Text reader (2.7 kB) * Calendar (1.5 kB) * Debug features (6.5 kB) * Edge overlay (2.3 kB, note that this is just the program code; edge overlay allocates much more memory when you actually use it!). * Built-in Lua libraries (10.5 kB total): ** IOLIB (3.3 kB) ** OSLIB (1.9 kB) ** STRLIB (5.3 kB) :Disabling these libraries cripple Lua scripting significantly as many Lua scripts assume you have all or some of these enabled. Basically this is almost the same as disabling Lua unless you are able to edit scripts to make them work without those libraries, but it's not currently possible to disable Lua completely from buildconf.inc. When you disable a feature from buildconf.inf, you completely and totally will not be able to use that feature, because it will not exist in your own private build of CHDK. Memory fragmentation You are probably familiar with hard disk fragmentation i.e. when over time, empty hard disk space tends to become scattered in small pieces all over the disk especially if the disk has at some point been rather full of data. Large files do not reside on the disc in one continous block, but are split in many smaller parts because no continuous space was available when saving the file. Fragmentation results in slow performance, because the drive needs to jump around a lot to read and write files. Similar things happen in your camera's RAM, but the consequences are worse. CHDK uses malloc() to ask for RAM from the camera's operating system. This fails unless a continuous area of RAM of the required size is found, even if there is enough available RAM in total. Think of the following scenario: :You happen to have 330kB available continuous RAM after power-on. You may then enable three camera features #1 #2 and #3 (in that order), each reserving 100 kB RAM. You then stop using features #1 and #3, freeing their RAM. You are now unable to use feature #4 which requires 200 kB of continuous memory (even though you have 230 kB free RAM total). If you would have enabled features in a different order (such as #2 #1 #3), you might have been able to start that 4th feature! :Note The above example assumes the camera's operating system always allocates RAM starting from the first byte of sufficient continuous free space it can find. I have no clue if this is actually the case, but it's not actually of much interest since the example is meant to be just a crude simplification of what may happen. Anyway, this is one reason why rebooting the camera may help in some memory related problems. For example, when experimenting with the large SunsetF15 Lua script on a570is I have noticed that size of largest continuous memory block available to malloc() decreases after starting and stopping the script a few times in a row. Especially if I enable some other RAM requiring features (like DNG), I'm able to get my camera to crash at half shoot if zebra kicks and allocates RAM for its buffers. I believe this is because of memory fragmentation. But this only happens if I start the script without first shooting a photo. This means that Canon's firmware allocates some RAM during the shoot process, but doesn't free it instantly after taking the photo. And when I start my script, Lua allocates a whole lot of things, then shoots and Canon allocates things, and when Lua frees RAM while stopping the script, memory is now fragmented! So, when using large Lua scripts in REC mode, it's adviceable do it this way if you think you may be low on RAM: # Power off. # Power on to REC mode. # Shoot one photo. # Start the script. # When done with the script, reboot. How to make Canon firmware use less RAM A small amount of memory (24 bytes for each JPEG has been experienced) is consumed for each JPEG (likely other files recognized by PLAY mode as well) in the Canon image directories (/DCIM/???CANON). This is not a problem if the card only has a few hundred images, but a large card (SDHC cards are available up to 16 GiB) can hold tens of thousands of small resolution images, easily achievable using timelapse scripts. 24 bytes per image for 20000 images equals 480000 bytes of RAM, which is much more than some cameras have available. Booting up with thousands of images on the card is slower than booting up with an empty card, because the Canon firmware scans the images. This is especially true when booting to PLAY mode, but somewhat affects booting to REC mode as well. It seems the camera can actually handle booting up with a large number of images on the card without crashing due to lack of RAM, things just just slow down a lot. But the camera updates (and resizes accordingly) its file tables during REC mode and a crash due to lack of RAM may occur if a script takes too many images without rebooting. Crashes have been experienced with a570is running a Lua timelapse script, but it is also possible that there is a 32k new files per session limit/bug in the camera firmware that caused this along with the fact that a new DCIM/???CANON directory is created when the image counter wraps around at 9999. This is not a bug, this is not even a feature. This is because the camera as well as EXIF don't have seperate directory and image counters internally but one large counter instead of which the lower four digits are used as the image number and the next higher three digits as a directory number when writing images to the mass storage. That's also why directory numbers start at 100. So when you are at image 9999 in directory 123 you have a true image number of 1239999 and the next image number consequently is 1240001 (1240000 is skipped to avoid an all zero image number). Sorry for putting this in here, but I don't know where else to put it. --Mandelschnitte 14:05, 25 July 2009 (UTC) (this hasn't been tested carefully, just some thoughts of what may have been happening). If you need to boot with lots of images on the card, booting up to PLAY mode before going switching to REC mode causes the camera certainly to read a full image table before doing anything else. It's possible that this requires more RAM than booting up to REC mode directly. But the opposite has actually been experienced very likely due to RAM fragmentation, which is a smaller problem when booting to PLAY mode and then switching to REC mode. To sum it up, to maximize free RAM: # Delete all images from the card (and other files the camera recognizes, DCIM/100CANON) and restart the camera. # If you can't delete images, hide them from Canon firmware by renaming the image files or directories. It's adviceable to reboot the camera after doing this if you managed to achieve this in CHDK (TODO: can Lua scripts rename directories? If so, create a script to rename these DCIM/???CANON folders. If not, create a mass rename script to rename all IMG????.JPG files). Available RAM by camera model Below is a table that intends to gives some primitive information to users and developers about the amount of RAM available on each model (in 8-bit bytes). Please add data for your camera if it's missing or incomplete. # Install the most recent 'complete' CHDK autobuild to an empty memory card (it must be empty, especially make sure there is no CHDK/CCHKD.CFG; otherwise your configuration settings will alter the result). # Start camera in REC mode (or start to PLAY mode and before doing anything else, enter REC mode). # Press ALT and then press MENU, navigate to the 'Miscellaneous stuff' menu and select 'Show memory info". Take extra care not to press anything before ALT. # Write down the two numbers and calculate their sum to the table below. Also fill in the other columns. : Notes * The "Free Memory" field is smaller than the actual amount of free RAM, if RAM is fragmented. This is why it's important not to do anything with the camera before taking the memory info. Maybe it's possible that RAM is already fragmented when CHDK starts, but this is all we can do for now, and the largest available block is the most interesting RAM measure there is if malloc() can't allocate in fragments. * The "free Memory" value may vary by as much as a few KB between restarts, even if the SD card and camera settings are identical. Category:CHDK internals Category:Development